Math - Stat - 491 - Fall 2014 - Notes - II

We will be closely following the book ”Essentials of Stochastic Processes”, 2nd Edition, by Richard Durrett, for the topic ‘Finite Discrete time Markov Chains’ (FDTM). These preliminary notes are for giving an overview, and for providing some guidance on what to aim at while studying this topic. An FDTM has a finite number of states on which it can rest at discrete times 1, 2, · · · . Usually the state at time n is denoted by Xn, and the possible set of states could be denoted by lower case symbols x, y · · · etc. If the chain is at state i at time n, it moves to state j at time n+1, with probability p(i, j). This could be stated alternatively as ‘the conditional probability of the chain being at state j at time n+ 1, given that it is at state i at time n is p(i, j).’ The key idea is that the probability of the chain being at state j at time n+ 1, does not depend on what happened before time n. Formally, we have the ‘Markov property’,